Display control apparatus, display control method, and program

ABSTRACT

A display control apparatus: in a virtual three-dimensional space in which is disposed a three-dimensional model of an object contained in a moving image picked up by performing successive imaging operations once by using an imaging device, draws a plurality of thumbnail images extracted from the moving image to the respective positions of the imaging device at the time when the thumbnail images were picked up and draws information that relates the thumbnail images in order in which they were picked up, thereby generating a bird&#39;s eye-display image which provides a bird&#39;s-eye-view drawing of the virtual three-dimensional space containing the three-dimensional model and the thumbnail images; and causes a display unit to display the bird&#39;s eye-display image.

BACKGROUND OF THE INVENTION

1. Field of the Invention

The present invention relates to a display control apparatus, a displaycontrol method, and a program.

2. Description of the Related Art

There are regular playback display and film roll display available as avideo display method for displaying moving images picked up andvideo-recorded with an imaging device such as a video camera. Theregular playback display method will display a video (x-axis and y-axisof an imaging plane) by reproducing its video-recorded moving imagessequentially from the beginning along a time axis t. The user can graspimaged contents by viewing the video thus reproduced regularly but in aconsiderable lapse of time. To reduce the time, a method is availablefor fast-forwarding the moving images in playback display but stillrequires a lapse of time to some extent in order to grasp the imagedcontents of the moving images as a whole.

On the other hand, the film roll display method will select a pluralityof frame images from among video-recorded moving images at apredetermined time interval and displaying the frame images as thumbnailimages sequentially in time series order (see Japanese PatentApplication Laid-Open No. 2007-134771 and Japanese Patent ApplicationLaid-Open No. 2008-166988, for example). According to the film rolldisplay approach, if an arbitrary one is selected from among thethumbnail images displayed in a list, it is possible also to display themoving images by regularly reproducing them starting from a playbackposition that corresponds to the thumbnail image. According to such filmroll display, the user can understand the imaged contents of the movingimages as a whole more easily than by the regular playback display.

SUMMARY OF THE INVENTION

However, in the case of the film roll display method, if the timeinterval at which thumbnail images are picked up is too short, the userneeds to view a lot of the thumbnail images and so spend a long time tograsp the imaged contents. On the other hand, if the time interval istoo long, an interrelationship among the thumbnail images is decreased,thus making it difficult for the user to grasp the imaged contents.Further, by the film roll display method, the thumbnail images are justarranged on a two-dimensional plane surface in time series order, sothat it is difficult to understand imaging situations such as theposition of an imaging device and the relative positional relationshipbetween the imaging device and a subject for photography at time ofphotographing.

The present invention has been made in view of the above issue, and itis desirable to provide a novel and improved display control apparatus,display control method, and program that can display the picked upcontents and imaging situations of picked up moving images in a way easyto understand.

According to an embodiment of the present invention, there is provided adisplay control apparatus including, a storage unit that stores a movingimage picked up by performing successive imaging operations once byusing an imaging device; a display image generation unit that draws aplurality of thumbnail images extracted from the moving image to therespective positions of the imaging device at the time when thethumbnail images were picked up in a virtual three-dimensional space inwhich a three-dimensional model of an object contained in the movingimage is disposed and draws information that relates the thumbnailimages in order in which they were picked up in the virtualthree-dimensional space, thereby generating a bird's eye-display imagewhich provides a bird's-eye-view drawing of the virtualthree-dimensional space containing the three-dimensional model and thethumbnail images; and a display control unit that causes a display unitto display the bird's eye-display image.

The storage unit may store movement track information which represents amovement track of the imaging device at the time when the imaging devicepicked up the moving image, as additional information of the movingimage and the display image generation unit may draw the thumbnailimages in the virtual three-dimensional space at the respectivepositions of the imaging device at the time when the thumbnail imageswere picked up based on the movement track information.

The information which relates the thumbnail images in order in whichthey were picked up may be a movement track line representing themovement track of the imaging device at the time when the imaging devicepicked up the moving image, and the display image generation unit maydraw the movement track line in the virtual three-dimensional space insuch a manner as to connect the thumbnail images in the order in whichthe thumbnail images were picked up based on the movement trackinformation.

The display image generation unit may draw the movement track lineconnecting the thumbnail images in various line types in accordance withcontents or imaging situations of the moving image between the thumbnailimages.

The display image generation unit may draw an alternative image whichrepresents the contents or imaging situations of at least some of thethumbnail images to the drawing positions of the thumbnail images inplace of the thumbnail images.

The display image generation unit may change a size of the thumbnailimages to be drawn in accordance with the drawing positions of thethumbnail images in the virtual three-dimensional space; and if the sizeof the thumbnail images to be drawn is a predetermined size or less, maydraw the alternative image in place of the thumbnail images.

The display image generation unit may draw metadata about the thumbnailimages in the virtual three-dimensional space in a condition where themetadata is related to the thumbnail images.

The display control apparatus further may include an image editing unitthat edits the moving image in response to user operations on thethumbnail images or the three-dimensional model in the bird'seye-display image shown on the display unit.

The display control apparatus further may includes an image editing unitthat generates, in response to specification of an angle in the bird'seye-display image shown on the display unit, the moving image as viewedat the specified angle.

The display control apparatus further may includes an image retrievalunit that retrieves the image containing the three-dimensional modelwhich is the same as or similar to the three-dimensional model selectedfrom the bird's eye-display image shown on the display unit or an objectcorresponding to the three-dimensional model.

According to another embodiment of the present invention, there isprovided a display control method including the steps of: drawing aplurality of thumbnail images extracted from a moving image picked up byperforming successive imaging operations once by using an imaging deviceto the respective positions of the imaging device at the time when thethumbnail images were picked up in a virtual three-dimensional space inwhich a three-dimensional model of an object contained in the movingimage is disposed and draws information that relates the thumbnailimages in order in which they were picked up in the virtualthree-dimensional space, thereby generating a bird's eye-display imagewhich provides a bird's-eye-view drawing of the virtualthree-dimensional space containing the three-dimensional model and thethumbnail images; and causing a display unit to display the bird'seye-display image.

According to another embodiment of the present invention, there isprovided a program for causing a computer to perform the steps of:drawing a plurality of thumbnail images extracted from a moving imagepicked up by performing successive imaging operations once by using animaging device to the respective positions of the imaging device at thetime when the thumbnail images were picked up in a virtualthree-dimensional space in which a three-dimensional model of an objectcontained in the moving image is disposed and draws information thatrelates the thumbnail images in order in which they were picked up inthe virtual three-dimensional space, thereby generating a bird'seye-display image which provides a bird's-eye-view drawing of thevirtual three-dimensional space containing the three-dimensional modeland the thumbnail images; and causing a display unit to display thebird's eye-display image.

According to the above configuration, in a virtual three-dimensionalspace in which is displayed a three-dimensional mode of an objectcontained in moving images picked up by a series of imaging operationsof an imaging device, a plurality of thumbnail images extracted from themoving images are drawn to their respective positions of the imagingdevice at times when the thumbnail images were picked up. Further, theinformation which relates the thumbnail images in order in which theyare picked up is drawn. As a result, a bird's eye-display image isgenerated which provides a bird's-eye-view drawing of the virtualthree-dimensional space that contains the three-dimensional model andthe thumbnail images. Furthermore, the bird's eye-display image isindicated on a display unit. Accordingly, a bird's eye-display image canbe displayed which suitably represents the picked up contents imagingsituations of moving images by using one image.

Thus, according to the present invention, it is possible to displaypicked up moving images in such a manner that their contents and imagingsituations can be grasped easily.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 shows a display device on which a bird's eye-display imageaccording to one embodiment of the present invention is shown;

FIG. 2 shows the display device on which a regular playback-displayimage according to the embodiment is shown;

FIG. 3 shows the display device on which a film roll display imageaccording to the embodiment is shown;

FIG. 4 shows an imaging device that displays a bird's eye-display imageaccording to a first embodiment of the present invention;

FIG. 5 is a block diagram showing a hardware configuration of theimaging device according to the embodiment;

FIG. 6 is a block diagram showing a functional configuration of theimaging device according to the embodiment;

FIG. 7 is a flowchart showing a method for displaying a bird'seye-display image according to the embodiment;

FIG. 8 shows a display example 1 of the bird's eye-display imageaccording to the embodiment;

FIG. 9 shows a display example 2 of the bird's eye-display imageaccording to the embodiment;

FIG. 10 shows a display example 3 of the bird's eye-display imageaccording to the embodiment;

FIG. 11 shows a display example 4 of the bird's eye-display imageaccording to the embodiment;

FIG. 12 shows a display example 5 of the bird's eye-display imageaccording to the embodiment;

FIG. 13 shows a display example 6 of the bird's eye-display imageaccording to the embodiment;

FIG. 14 shows a display example 7 of the bird's eye-display imageaccording to the embodiment;

FIG. 15 shows an information processing apparatus that displays a bird'seye-display image according to a second embodiment of the presentinvention;

FIG. 16 is a block diagram showing a hardware configuration of theinformation processing apparatus according to the embodiment;

FIG. 17 is a block diagram showing a functional configuration of theinformation processing apparatus according to the embodiment;

FIG. 18 shows a display example 1 of the bird's eye-display imageaccording to the embodiment;

FIG. 19 shows a display example 2 of the bird's eye-display imageaccording to the embodiment;

FIG. 20 shows a display example 3 of the bird's eye-display imageaccording to the embodiment;

FIG. 21 shows a display example 4 of the bird's eye-display imageaccording to the embodiment;

FIG. 22 shows a display example 5 of the bird's eye-display imageaccording to the embodiment; and

FIG. 23 shows a display example 6 of the bird's eye-display imageaccording to the embodiment.

DETAILED DESCRIPTION OF THE EMBODIMENTS

Hereinafter, preferred embodiments of the present invention will bedescribed in detail with reference to the appended drawings. Note that,in the specification and the appended drawings, structural elements thathave substantially the same function and structure are denoted with thesame reference numerals, and repeated explanation of these structuralelements is omitted.

A description will be given in the following order.

1. Outline of bird's eye display2. First embodiment (Bird's eye display at the time of imaging byimaging device)3. Second embodiment (Bird's eye display at the time of playback byinformation processing apparatus)

1. Outline of Bird's Eye Display

First, a description will be given of the outline of bird's eye displayaccording to the first embodiment of the present invention withreference to FIG. 1. FIG. 1 shows an example of a display device 1, onwhich a bird's eye-display image 3 according to the present embodimentis displayed.

Bird's eye display according to the present embodiment is a videodisplay approach for facilitating the user to grasp the picked upcontents and imaging situations of a video (moving images) picked upwith an imaging device when the video is being displayed on the displaydevice 1.

As shown in FIG. 1, on a display screen 2 of the display device 1, abird's eye-display image 3 is displayed. The bird's eye-display image 3is a still image that schematically represents the contents of movingimages picked up by a series of imaging operations of the imagingdevice. A series of imaging operations refer to, for example, seriallypicking up the moving images of a subject from a point in time when animaging start button on the imaging device is pressed to a point in timewhen an imaging stop button on it is pressed, that is, “1rec”. Byperforming the series of imaging operations, moving image data isobtained as one imaging unit and then, usually recorded in a recordingmedium as one file. The bird's eye-display image 3 functions as an indeximage which is used to grasp the picked up contents and imagingsituations of the data of an entirety of moving images picked up by theseries of imaging operations.

Specifically, as shown in FIG. 1, the bird's eye-display image 3provides a bird's eye drawing of a virtual three-dimensional space 6containing three-dimensional models 4 of an object in a picked up movingimage and a plurality of thumbnail images 5 extracted from the movingimage. The virtual three-dimensional space 6 schematically represents athree-dimensional space (subject in an imaging range) of a realityimaged by the imaging device. In the virtual three-dimensional space 6,the three-dimensional models 4 of the object contained in the picked upmoving image are disposed to the corresponding positions in athree-dimensional space of the reality.

The object means a target (that is, some subjects which were present inthe imaging range) contained in the picked up moving image. An objectrepresented as the three-dimensional model 4 may be, for example, anarbitrary tangible entity such as stuff, a person, a vehicle, or abuilding but should preferably have at least a certain size in order tobe displayed as something which can be understood easily in the virtualthree-dimensional space 6. The three-dimensional model 4 of such anobject may be disposed as almost the same shape, to almost the sameposition, and in almost the same direction as those of an imaged objectthat was present in the three-dimensional space of the reality. In anexample of the bird's eye-display image 3 of FIG. 1, the gymnasium of anelementary school is represented as the virtual three-dimensional space6, while as the three-dimensional model 4 of the object, a plurality oftables 4 a and a stage 4 b present at the time of imaging are displayed.

Furthermore, in the virtual three-dimensional space 6 of the bird'seye-display image 3, a plurality of the thumbnail images 5 extractedfrom the moving image are drawn to their respective positions and intheir respective directions of the imaging device at the time when thethumbnail images 5 were picked up. The thumbnail images 5 are frameimages (still images) extracted from the data of a moving image pickedup and video-recorded by a series of imaging operations of the imagingdevice. In one method for extracting thumbnail images from a movingimage, a predetermined number of the thumbnail images 5 may be extractedfrom the moving image at a predetermined time interval. Further,thumbnail images picked up without moving the imaging device from afixed position or thumbnail images having a good image quality may beselected and extracted from a moving image. It is thus possible to limitthe number of the thumbnail images 5 to be displayed in the bird'seye-display image 3 to an appropriate number. Thus, a lot of thethumbnail images 5 will be drawn in the bird's eye-display image 3,thereby preventing complicated and redundant display.

In the example of FIG. 1, five thumbnail images 5 a, 5 b, 5 c, 5 d, and5 e are drawn in the virtual three-dimensional space 6 in order in whichthey were picked up. The thumbnail images 5 a-5 e are disposed topositions and in directions that respectively correspond to the actualpositions and directions of the imaging device when the thumbnail images5 a-5 e were picked up. For example, the top thumbnail image 5 a (firstthumbnail image 5 a) of the moving image is disposed to a position thatcorresponds to an imaging starting position of the imaging device, thatis, toward the user and to the left side in the virtualthree-dimensional space 6. Further, the thumbnail image 5 a is disposedin a direction (direction toward the three-dimensional model 4 b of thestage from the imaging starting position) of the imaging device at thetime when the thumbnail image 5 a was picked up.

Further, the thumbnail images 5 are drawn in perspective in the virtualthree-dimensional space 6 in such a manner that they may be largertoward front and smaller toward the back in the virtualthree-dimensional space 6. For example, the first thumbnail image 5 aand the fifth thumbnail image 5 e were picked up from an actual imagingposition toward front and so drawn as expanded toward you in the virtualthree-dimensional space 6. On the other hand, the third thumbnail image5 c was picked up from an actual imaging position toward the back most(toward the stage) and so drawn as shrunk toward the back in the virtualthree-dimensional space 6.

In such a manner, the thumbnail images 5 are drawn in the virtualthree-dimensional space 6 in such a manner that their positions,directions, and sizes may correspond to their respective positions andin their respective directions of the imaging device at the time whenthey were picked up.

Moreover, in the virtual three-dimensional space 6, a movement trackline 7 of the imaging device is drawn in such a manner as tointerconnect the plurality of thumbnail images 5 sequentially. Themovement track line 7 is one example of information that interrelatesthe plurality of thumbnail images 5 in order in which they were pickedup and represents a movement track of the imaging device as it picked upa moving image actually. The movement track line 7 interconnects theplurality of thumbnail images 5 along time series in order in which theywere picked up and is drawn along an actual movement path (that is,imaging positions) of the imaging device. By drawing such a movementtrack line 7, it is possible to properly represent the interrelationshipand pickup positions of the thumbnail images 5. Accordingly, based onthe movement track line 7, the user can easily understand the order inwhich the thumbnail images 5 were picked up and the movement track ofthe imaging device as it picked up the moving image. It is to be notedthat although not described in detail, the movement track line 7 isdrawn based on imaging position information obtained with varioussensors mounted on the imaging device.

In such a manner, in the bird's eye-display image 3 and in the virtualthree-dimensional space 6 wherein the three-dimensional models 4 of anobject contained in a picked-up moving image are disposed, a pluralityof the thumbnail images 5 extracted from the moving image are drawn totheir respective positions and in their respective directions of theimaging device at the time of imaging. Then, the bird's eye-displayimage 3 which provides a bird's-eye-view drawing of the virtualthree-dimensional space 6 containing the thumbnail images 5 and thethree-dimensional models 4 of such an object is generated and displayedon the display screen 2 of the display device 1. Thus, in the virtualthree-dimensional space 6, the bird's eye-display image 3 may provide abird's eye display of the thumbnail images 5 of an actually picked-upmoving image and the three-dimensional models 4 of an object whichappears in the moving image in a condition where they are interrelated.Hereinafter, this display approach is referred to as “four-dimensionaltime-space bird's eye display”.

By viewing the bird's eye-display image 3 given in such four-dimensionalbird's eye display, the user can simultaneously grasp a depth axis z anda photographing time axis t as well as the x- and y-axes of the imagingplane at the time when moving image data was obtained. That is, the usercan easily grasp not only the contents of a moving image but also itsimaging situations based on the positional relationship between thethree-dimensional models 4 and the thumbnail images 5 in the bird'seye-display image 3, the layout, directions, and sizes of the thumbnailimages 5, and the movement track line 7. Accordingly, the user can viewthe bird's eye-display image 3 which provides index display of a movingimage, thereby easily understanding where and how the moving image waspicked up.

It is to be noted that the display device 1 for displaying the bird'seye-display image 3 may be either a two-dimensional display (x, y) or athree-dimensional display (x, y, z).

Further, the method for generating the three-dimensional model 4 of anobject is not limited to any predetermined method and may be any methodpublicly known, coming in, for example, the following methods (a)-(d).

(a) Method for Generating the Three-Dimensional Model 4 by UtilizingVarious Sensors Mounted on the Imaging Device

For example, the imaging device may be equipped with a distance sensorwhich measures a distance between the imaging device and an object atthe time of imaging and a movement track sensor which measures amovement track or posture of the imaging device at the time of imaging.The movement track sensor may be, for example, an angular velocitysensor, an acceleration sensor, a magnetic sensor, a global positioningsystem (GPS) sensor, etc. The display control apparatus such as theimaging device or the information processing apparatus uses such varioussensors, to measure the distance between the imaging device and anobject as well as the movement track or posture of the imaging device ata time when a moving image was picked up. Then, the display controlapparatus analyzes the measurement data and moving image data, toestimate the shape, the position, and the direction of the objectpresent in an imaged space, thereby generating its three-dimensionalmodel 4.

(b) Method for Automatically Extracting the Three-Dimensional Model 4from Moving Image Data by Using Software

Software (program) for generating a three-dimensional model is installedto the display control apparatus such as the imaging device or theinformation processing apparatus. The software analyzes the data of amoving image and identifies and extracts an object contained in a videogiven by the moving image data, to estimate its shape, position, andsize, thereby generating the three-dimensional model 4. It is to benoted that the display control apparatus may be arranged to acquire thedata of the three-dimensional model 4 similarly generated by any otherdevice from the device via a recording medium or a network.

(c) Method for Manually Generating the Three-Dimensional Model 4

The user manually generates a three-dimensional model 4 as viewing animage displayed based on reproduced moving image data. The manualreproduction of the three-dimensional model 4 may be carried out in thedisplay control apparatus such as an imaging device or an informationprocessing apparatus or any other devices. In the case of the latter,the display control apparatus may acquire information about thethree-dimensional model 4 via a network from these any other devices orrecording media.

(d) Combination of Three-Dimensional Models 4 Generated by the Methodsof (a)-(c)

The display control apparatus combines three-dimensional models 4generated by the methods of (a)-(c), to generate a finalthree-dimensional model 4 to be disposed in the virtualthree-dimensional space 6.

A description has been given of the methods for generating thethree-dimensional model 4 of an object contained in a moving image. Itis to be noted that the three-dimensional model 4 to be drawn in thevirtual three-dimensional space 6 may be a faithful representation orsimplified representation of an actual object or such a simplifiedthree-dimensional model 4 from which unnecessary entities have beenomitted. For example, in the bird's eye-display image 3, thethree-dimensional model 4 may be displayed simply in black and white. Itis thus possible to emphasize the thumbnail images 5 disposed aroundthat three-dimensional model 4 so that the thumbnail images 5 may beviewed easily. It is to be noted that in the bird's eye-display image 3,rather than drawing none of the three-dimensional model 4 at all in thevirtual three-dimensional space 6, only the thumbnail images 5 may bedrawn or, without drawing the three-dimensional model 4, the thumbnailimages 5 and the later-described movement track line 7 or metadata 16may be drawn instead.

Further, the display control apparatus can select a position where thethree-dimensional model 4 is to be drawn in the bird's eye-display image3, in accordance with the imaging time in a moving image specified bythe user. For example, by selecting only one time, all of times, or aperiodic time for each three-dimensional model 4, its drawing positionand whether it is to be drawn can be controlled.

Further, although not only horizontal movement but also verticalmovement of the imaging position can be represented in the bird'seye-display image 3, three-dimensional models of an object inconvenientto look down at (for example, a floor or a ceiling) may be omitted indrawing as appropriate.

Further, the display control apparatus is capable of arbitrarilychanging a viewpoint (viewpoint in four-dimensional time-space bird'seye display) that serves as a reference in providing bird's eye displayof the above bird's eye-display image 3, in accordance with userselection. For example, the display control apparatus can analyze amoving image to thereby obtain an optimal viewpoint to grasp contents ofthe moving image as a whole and present the user with it as arecommendable viewpoint.

A comparison will be given between the bird's eye-display image 3 (FIG.1), an image 8 by regular playback display (FIG. 2), and an image 9 byfilm roll display (FIG. 3) with reference to FIGS. 2 and 3.

As shown in FIG. 2, regular playback display is a regular display methodof reproducing videos (x- and y-axes of the imaging plane) of avideo-recorded moving image sequentially along the time t-axis. In suchregular playback display, only one of the images 8 being reproduced isdisplayed, thus taking a long time to grasp the contents of the movingimage as a whole.

Further, as shown in FIG. 3, film roll display is a method of extractinga plurality of thumbnail images (frame images) 11 a-11 e at apredetermined time interval from a video-recorded moving image andsequentially displaying the images 1 a-1 e in time series order. Suchfilm roll display provides easy-to-grasp imaged contents as compared tothe above regular playback display. However, if the time interval atwhich the thumbnail images 11 a-11 e are extracted is too short, aplurality of the thumbnail images 11 a-11 e are displayed at a time andso complicated, thus taking a long time to grasp the imaged contents. Onthe other hand, if the time interval is too long, an interrelationshipamong the thumbnail images 11 a-11 e deteriorates, thus disablinggrasping the imaged contents.

Meanwhile, as shown in FIG. 1, in the bird's eye-display image 3according to the present embodiment, it is possible not only tosimultaneously grasp the x-axis, y-axis, and depth z-axis of the imagingplane and the photographing time t-axis of a picked-up moving image butalso to grasp photographing order, imaging positions, and directions ofthe thumbnail images 5. Therefore, the user can grasp where and how hepicked up that moving image, thus easily understanding the imagedcontents and the imaging situations.

The bird's eye-display image 3 by use of four-dimensional time-spacebird's eye display has been outlined as above. Next, a description willbe given of suitable embodiments of the display control apparatus anddisplay control method for displaying the bird's eye-display image 3.

2. First Embodiment

First, a description will be given of the display control apparatus andmethod according to the first embodiment of the present invention.

As shown in FIG. 4, the first embodiment may be an example in which thedisplay control apparatus of the present invention has been applied toan imaging device 10 (for example, video camera). As a real timeapplication in imaging by the imaging device 10, the bird's eye-displayimage 3 is displayed on a display device 130 equipped to the imagingdevice 10. For example, if the user would like to confirm thevideo-recorded contents of a 1rec moving image as or after picking it upand video-recording it with the imaging device 10, the bird'seye-display image 3 about data of the moving image will be displayed onthe display device 130.

[Hardware Configuration of Imaging Device]

First, a description will be given of a hardware configuration of theimaging device 10 according to the present embodiment in detail withreference to FIG. 5. FIG. 5 is a block diagram showing the hardwareconfiguration of the imaging device 10 according to the presentembodiment.

As shown in FIG. 5, the imaging device 10 according to the presentembodiment schematically includes an imaging unit 110, a signalprocessing unit 120, a display device 130, a recording device 140, acontrol unit 150, an operation unit 160, a distance sensor 170, and amovement track sensor 172.

The imaging unit 110 images a subject and outputs an analog imagesignal. The imaging unit 110 includes an imaging optical system 111, animaging element 112, a timing generator 113, and an optical elementdrive unit 114.

The imaging optical system 111 includes various lenses such as a focuslens and a zoom lens, an optical filter that removes unnecessarywavelengths, and an optical element such as an aperture diaphragm. Lightmade incident from the subject passes through the optical elements inthe imaging optical system 111 up to the imaging element 112. Theimaging element 112 includes a solid-state imaging element such as, forexample, a charge coupled device (CCD) or a complementary metal oxidesemiconductor (CMOS). The imaging element 112 conducts photoelectricconversion on the light guided via the imaging optical system 111 andoutputs an electric signal (analog image signal) that represents apicked up image.

To the imaging optical system 111, an actuator (not shown) ismechanically connected which is a drive mechanism for adjusting theaperture diaphragm and moving the focus lens. The actuator is connectedalso to the optical element drive unit 114, which controls drivingthereof. The optical element drive unit 114 controls the driving of theoptical elements of the imaging optical system 111 in accordance withinstructions from the later-described central processing unit (CPU) 151.The timing generator (TG) 113 generates operations pulses necessary forthe imaging element 112 in accordance with the instructions from the CPU151. For example, the TG70 generates various pulses such as four-phasepulses and field-shift pulses necessary for vertical transfer andtwo-phase pulses and shutter pulses necessary for horizontal transferand supplies the pulses to the imaging element 112. The TG70 enablesdriving the imaging element 112 (electronic shutter function).

The image signal output from the imaging element 112 is input to thesignal processing unit 120. The signal processing unit 120 performspredetermined signal processing on the image signal supplied from theimaging element 112 and the post-signal processing image signal to thedisplay device 130 and the recording device 140. The signal processingunit 120 includes an analog signal processing unit 121, ananalog/digital (A/D) conversion unit 122, and a digital signalprocessing unit 123.

The analog signal processing unit 121 is so-called an analog front endfor performing preprocessing on the image signal. For example, theanalog signal processing unit 121 performs correlated double sampling(CDS) processing and gain processing by use of a programmable gainamplifier (PGA) on the electric signal as a picked up image output fromthe imaging element 112. Then, the analog signal processing unit 121supplies the A/D conversion unit 122 with the image signal obtained as aresult of the preprocessing. The A/D conversion unit 122 converts thesupplied analog image signal into a digital image signal and supplies itto the digital signal processing unit 123. The digital signal processingunit 123 performs digital signal processing such as, for example, gammacorrection processing and white balance processing on the supplieddigital image signal, thus generating a video signal of the picked upimage. The digital signal processing unit 123 outputs the thus generatedvideo signal of the picked-up image data to the display device 130 andthe recording device 140.

The display device 130 includes a liquid crystal display (LCD) or anorganic EL display. The display device 130 displays various kinds ofimage data supplied thereto. For example, the display device 130displays the data of images in the middle of pick-up as it is beingsupplied from the signal processing unit 120 in real time duringimaging. This permits the user to carry out photographing as viewing theimages being picked up by the imaging device 10. Further, if the data ofpicked up images recorded in the recording device 140 is reproduced, thedisplay device 130 displays the reproduced image data input from therecording device 140. This permits the user to confirm the contents ofpicked-up image data recorded in the recording device 140. It is to benoted that the display device 130 displays the above bird's eye-displayimage 3 (see FIG. 1), whose details will be described later.

The recording device 140 records in a recording medium the various kindsof data such as data of the thus picked up images and its metadata. Therecording device 140 can include an arbitrary recording device such asan optical disc drive or a hard disk drive (HDD). The recording mediumto be used may be a removable medium such as, for example, an opticaldisc, a semiconductor memory such as a memory card, or a hard disk. Theoptical disc may be, for example, a blu-ray disc, a digital versatiledisc (DVD), or a compact disc (CD). The recording medium to be used mayhave been built in the imaging device 10.

The control unit 100 includes a microcontroller and controls operationsof the imaging device 10 as a whole. The control unit 100 includes, forexample, the CPU 151, an EEPROM 152, a read only memory (ROM) 153, and arandom access memory (RAM) 155. It is to be noted that the EEPROM standsfor “electrically erasable programmable ROM”.

The CPU 151 controls the TG 113 and the optical element drive unit 114in the imaging unit 110, thus controlling imaging processing by theimaging unit 110. For example, the CPU 151 conducts automatic exposurecontrol by adjusting the aperture diaphragm of the imaging opticalsystem 111 and setting an electronic shutter speed of the imagingelement 112 and a gain of an AGC in the analog signal processing unit121. Further, the CPU 151 conducts auto-focus control to automaticallyfocus the lens on the subject by, for example, driving the focus lens inthe imaging optical system 111. Further, the CPU 151 controls processingto record and reproduce picked-up image data by using the recordingdevice 140. Further, the CPU 151 conducts display control to displayvarious kinds of display data on the display device 130. It is to benoted that the CPU 151 performs processing to generate the bird'seye-display image 3 (see FIG. 1) and display control processing todisplay the bird's eye-display image 3 on the display device 130, whosedetails will be described later.

In the ROM 153 in the control unit 100, a program is stored which isused to cause the CPU 151 to perform various kinds of controlprocessing. The CPU 151 operates based on the program, to performarithmetic operations and control processing necessary for the abovevarious kinds of control as using the RAM 154. The program can be storedbeforehand in a storage device (for example, the ROM 153) built in theimaging device. Further, the program may be provided to the imagingdevice in a condition where it is stored on a removable recording mediumsuch as a CD, DVD, blu-ray disc, or memory card or may be downloaded tothe imaging device via a network such as a LAN or the Internet.

The operation unit 160 and the display device 130 function as a userinterface. The operation unit 160 includes an operation button or atouch panel and outputs designated information to the control unit 100in accordance with user operations. It is to be noted that the operationunit 160 and the display device 130 may each include a device separatefrom the imaging device 10.

The distance sensor 170 is used to measure a distance between a subjectto be imaged and the imaging device 10. The distance sensor 170 enablesmeasuring the distance between an object contained in an imaging rangeas the subject and the imaging device 10 during imaging by the imagingdevice 10. On the other hand, the movement track sensor 172 is used tomeasure a position, a movement track, and a posture of the imagingdevice 10 during imaging and may include, for example, an angularvelocity sensor, an acceleration sensor, a magnetic sensor, or a GPSsensor. The distance sensor 170 and movement track sensor 172 are usedto draw the three-dimensional model 4 and the movement track line 7present in the bird's eye-display image 3, whose details will bedescribed later.

[Functional Configuration of Imaging Device]

Next, a description will be given in detail of the functionalconfiguration of the imaging device 10 according to the presentembodiment with reference to FIG. 6. FIG. 6 is a block diagram showingthe functional configuration of the imaging device 10 according to thepresent embodiment. It is to be noted that FIG. 6 schematically showsthe functional units that are used to display the bird's eye-displayimage 3, which features the present embodiment.

As shown in FIG. 6, the imaging device 10 includes a storage unit 20, animaging processing unit 40, a preprocessing unit 50, a display imagegeneration unit 60, the display control unit 70, the display device 130,the operation unit 160, the distance sensor 170, and the movement tracksensor 172. Of these, the preprocessing unit 50, the display imagegeneration unit 60, and the display control unit 70 are realized by, forexample, software installed in the imaging device 10. That is, forexample, the CPU 151 shown in FIG. 5 operates in accordance with theprogram stored in the ROM 153 etc., thereby functioning as thepreprocessing unit 50, the display image generation unit 60, and thedisplay control unit 70. It is to be noted that the program can beprovided to the imaging device 10 via a recording medium or a network.Next, a description will be given in detail of each of the units.

The storage unit 20 stores various information pieces 21-31 such asmoving image data 21 and thumbnail image information 26. The storageunit 20 includes, for example, the recording device 140 or the EEPROM152 shown in FIG. 5.

The imaging processing unit 40 images the subject and obtains the movingimage data 21. The imaging processing unit 40 includes, for example, theimaging unit 110 and the signal processing unit 120 shown in FIG. 5. Theimaging processing unit 40 records the obtained moving image data 21 inthe storage unit 20.

The preprocessing unit 50 performs preprocessing to generate variouskinds of information and data necessary for generating the bird'seye-display image 3 such as shown in FIG. 1. The preprocessing unit 50includes a distance estimation unit 51, a speech estimation unit 52, amovement track estimation unit 53, a three-dimensional shape estimationunit 54, a thumbnail image extraction unit 55, an alternative imagegeneration unit 56, an imaging information link unit 57, and a subjectestimation unit 58.

The distance estimation unit 51 performs processing to estimate adistance relation between the imaging device 10 and an object. Asdescribed above, the imaging device 10 is equipped with the distancesensor 170 which utilizes, for example, a laser or a sound ormillimeter-wave radar. The distance sensor 170 measures the distancebetween the imaging device 10 and an object present in the imaging rangeas the subject, during imaging by the imaging device 10. The distanceestimation unit 51 estimates the distances from the imaging device 10 toa plurality of the objects present in the imaging space and thedistances between the objects based on the distance information measuredby the distance sensor 170 and then results of analysis on the movingimage data 24. Then, the distance estimation unit 51 generates adistance map 22 which indicates the positional relations between theimaging device 10 and the objects at the time of imaging based on theestimated distance information and saves the distance map 22 in thestorage unit 20. It is to be noted that to measure the distance from theimaging device to a target, a ranging method is available based onstereopsis (binocular stereopsis) using a plurality of cameras (imagingunits). The distance estimation unit 51 can estimate the distance fromthe imaging device 10 to a plurality of objects present in an imagingspace and the distance between the objects by using the ranging methodbased on the stereopsis (binocular stereopsis).

The speech estimation unit 52 analyzes audio data added to the movingimage data 21, to detect an image position (which corresponds to animaging time) at which a sound of interest in the moving image has beenrecorded. The sound of interest refers to, for example, a sound relatedto the contents of the moving image such as, for example, words (speech)spoken by a person or a sound effect of objects, excluding sounds suchas noise and environmental sounds. The speech estimation unit 52generates speech information 23 that represents an image position atwhich a meaningful sound is recorded and saves it in the storage unit20. It is to be noted that although a sound acquisition unit (microphoneetc.) to acquire sounds at the time of imaging has been omitted in theblock diagram of the imaging device shown in FIG. 5, the imaging device10 can include the sound acquisition unit.

The movement track estimation unit 53 estimates a movement track and aposture of the imaging device 10 at a time when the moving image hasbeen picked up based on results of the analysis of the moving image data21 and the measurement data by the movement track sensor 172. Asdescribed above, the imaging device 10 is equipped with the distancesensor 170. The distance sensor 170 may be, for example, an angularvelocity sensor, an acceleration sensor, a magnetic sensor, or a GPSsensor. The angular velocity sensor and the acceleration sensor detectan angular velocity and an acceleration as the imaging device 10 hasmoved during imaging, respectively. The magnetic sensor detects amagnetic field around the imaging device 10, thereby detecting adirection of the imaging device 10. The GPS sensor detects positioninformation (latitude and longitude) of the imaging device 10. Themovement track sensor 172 supplies the movement track estimation unit 53with measurement data such as the thus detected angular velocity,acceleration, variation in magnetic field, and imaging position of theimaging device 10. The movement track estimation unit 53 analyzes suchmeasurement data and the above moving image data 21, thereby estimatinga movement track and a posture of the imaging device 10 at the time whenthe moving image has been picked up. Then, the movement track estimationunit 53 saves movement track information 24 that indicated the estimatedmovement track and posture of the imaging device 10, in the storage unit20.

The three-dimensional shape estimation unit 54 estimatesthree-dimensional shapes and positions of objects (for example, thetable and the stage in FIG. 1) contained in the moving image based onthe distance map 22 and the movement track information 24 about theimaging device 10. The objects contained in the moving image refer tothe objects that have been present as a subject in the imaging range atthe time of imaging. Then, the three-dimensional shape estimation unit54 stores object information 25 that represents the estimatedthree-dimensional shapes and positions of the objects, in the storageunit 25. The three-dimensional shape estimation unit 54 may generate thethree-dimensional models 4 of the respective objects based on theestimated three-dimensional shapes and positions of the objects and saveinformation of the three-dimensional models 4 in the storage unit 20.

The thumbnail image extraction unit 55 extracts a plurality of thethumbnail images 5 from the data of the moving image picked up by theimaging device 10. The thumbnail images 5 are frame images thatrepresent feature images of that moving image. The thumbnail imageextraction unit 55 may automatically extract a predetermined number ofthe thumbnail images 5 from the moving image at a predetermined timeinterval or selectively extract the thumbnail images 5 picked up in acondition where the device is stopped or those that have a good imagequality. The thumbnail image extraction unit 55 saves the extractedplurality of thumbnail images 5 as they are or information thatindicates the corresponding imaging positions (imaging times) asthumbnail image information 26, in the storage unit 20.

The alternative image generation unit 56 generates an alternative imagethat represents the contents or imaging situation of the thus extractedthumbnail images 5. The alternative image will be displayed in place ofthe thumbnail images 5 in the bird's eye-display image 3, coming in anicon, an illustration, a still image, or a text, for example. Theimaging situation of the thumbnail images 5 includes, for example, adirection of the imaging device 10 at the time of picking up thethumbnail images 5, camera's preset values such as a zoom, a focus, anexposure, and an image quality, a movement speed of the imaging device,an operating condition of the imaging device 10, and a camera shake. Thecontents (imaged contents) of the thumbnail images 5 are informationabout subjects contained in the thumbnail images 5, coming in names,identification information, operations, sizes, and imaging environmentsof the subjects, for example. The alternative image generation unit 56can convert the thumbnail images 5 into icons and illustrations, therebygenerating the icons and the illustrations (for example, illustrationsof the faces of persons contained in the thumbnail images 5) of thecontents of the thumbnail images 5. The alternative image generationunit 56 saves the thus generated alternative image 27 in the storageunit 20.

The imaging information link unit 57 relates imaging informationcontaining imaging person information 30 and imaging device information31 to the moving image data 21, thus generating link information 28 ofthe imaging information. The imaging person information 30 represents aperson who imaged the moving image, such as the name, identificationinformation, and behavior of the person. The imaging device information31 is related to the imaging device 10 which has given the moving imagedata 21, thus representing the imaging conditions such as themanufacturer's name, model name, identification information, and camerainstallation values of the imaging device 10, the operating conditionsof the imaging device 10, the type of videos, the format of images, andthe operations, for example. The imaging person information 30 and theimaging device information 31 may be generated based on user operationson the operation unit 160 or automatically by the imaging device 10.

The imaging information including such imaging person information 30 andimaging device information 31 represents imaging conditions under whichthe moving image data 21 has been obtained through imaging and functionsas identification information or attribute information of the movingimage data 21. Therefore, by relating such imaging information to themoving image data 21, it is possible to identify moving image data andmore accurately grasp the contents of the moving image data 21.Accordingly, the imaging information link unit 57 relates the imaginginformation and the moving image data 21 to generate the linkinformation 28 of the imaging information and saves it in the storageunit 20. It is to be noted that the imaging information may be relatedto the 1rec moving image data 21 as a whole or for each predeterminedimage unit (for example, each frame) thereof, depending on its contents.In the case of the latter, it is possible to relate the imaginginformation to each of the thumbnail images 5 and, therefore, displaythe thumbnail images 5 and metadata indicating their contents or imagingsituations in the bird's eye-display image 3 (see FIG. 10).

The subject estimation unit 58 analyzes the moving image data 21 toestimate a subject contained in the picked up moving image, therebygenerating subject information 29. The subject information 29 containsthe name, behavior, size, and imaging environments of the subjectcontained in the moving image. The subject estimation unit 58 saves thegenerated subject information 29 in the storage unit 20.

The display image generation unit 60 generates the bird's eye-displayimage 3 based on the moving image data 21 and the various informationpieces generated by the preprocessing unit 50, that is, data pieces22-29. Specifically, the display image generation unit 60 draws thevirtual three-dimensional space 6 in which the three-dimensional model 4of the object contained in the moving image data 21 is disposed based onthe above distance map 22, movement track information 24, objectinformation 25, etc. Then, the display image generation unit 60 drawsthe plurality of thumbnail images 5 extracted from the moving image data21 at their respective imaging positions in the virtualthree-dimensional space 6 based on the movement track information 24,the thumbnail image information 26, etc. Further, the display imagegeneration unit 60 draws the movement track line 7 that interrelates thethumbnail images 5 in order in which they were picked up based on themovement track information 24 etc. As a result, as shown in FIG. 1, thebird's eye-display image 3 is generated which provides a bird's-eye-viewdrawing of the virtual three-dimensional space 6 containing thethree-dimensional model 4 and the thumbnail images 5.

The display image generation unit 60 identifies positions (imagingpositions of the thumbnail images 5) and directions of the imagingdevice 10 at a time when it has picked up the thumbnail images 5 basedon the movement track information 24 and draws the thumbnail images atthe positions and in the directions respectively in the virtualthree-dimensional space 6. In this case, the display image generationunit 60 changes the sizes of the thumbnail images 5 and thethree-dimensional model 4 in accordance with positions at which they aredrawn in the virtual three-dimensional space 6. For example, thethumbnail images 5 will be small if they are drawn backward in thevirtual three-dimensional space 6 and will be large if they are drawnforward in the virtual three-dimensional space 6. Thus, the virtualthree-dimensional space 6 can be displayed as if viewed from anobliquely upward viewpoint so that the imaging situations may be graspedeasily.

It is to be noted that the display image generation unit 60 may relatesthe generated bird's eye-display image 3 to the moving image data 21 andsave it in the storage unit 20. This eliminates the necessity ofregenerating the bird's eye-display image 3 of the same moving imagedata 21. Further, the display image generation unit 60 can draw variouskinds of additional information in the bird's eye-display image 3 basedon the speech information 23, the alternative image 27, the linkinformation 28 of the imaging information, or the subject information29, whose details will be described later.

The display control unit 70 controls the display device 130 (forexample, LCD) so that it may display the bird's eye-display image 3generated by the display image generation unit 60. This may cause thedisplay device 130 to display the bird's eye-display image 3. It is tobe noted that display control by the display control unit 70 andgeneration of the bird's eye-display image 3 by the display imagegeneration unit 60 may be performed in response to user operations onthe operation unit 160 or automatically performed in accordance with thesetting.

[Display Control Flow]

Next, a description will be given of a method for displaying the bird'seye-display image 3 in the imaging device 10 according to the presentembodiment with reference to FIG. 7. FIG. 7 is a flowchart showing amethod for displaying the bird's eye-display image 3 according to thepresent embodiment.

As shown in FIG. 7, first the CPU 151 in the imaging device 10 analyzesthe moving image data 21 stored in the storage unit 20 shown in FIG. 6and data measured by the distance sensor 170 and the moving focus sensor172 (S10). Thus, the various information and data pieces 22-29 such asshown in FIG. 6 are obtained.

Subsequently, the CPU 151 generates the three-dimensional models 4 of aplurality of objects present in the picked up moving image based on thedistance map 22, the moving track information 24, and the objectinformation 25 (S12). Further, the CPU 151 extracts a plurality of thethumbnail images 5 from the moving image data 21 in accordance with thepredetermined extraction criterion (S14).

Then, the CPU 151 draws the virtual three-dimensional space 6 in whichthe three-dimensional models 4 of the objects generated at S12 aredisposed to positions that correspond to actual positions of the objects(S16). Subsequently, the CPU 151 draws the plurality of thumbnail images5 extracted at S14 in the virtual three-dimensional space 6 to thepositions and in the directions of the imaging device 10 at the timewhen the thumbnail images 5 were picked up (S18). Further, the CPU 151draws the movement track line 7 that relates the thumbnail images 5 inorder in which they were picked up based on the movement trackinformation 24 etc. (S20). It is thus possible to generate the bird'seye-display image 3 that provides a bird's-eye-view drawing of thevirtual three-dimensional space 6 containing the three-dimensionalmodels 4, the thumbnail images 5, and the movement track line 7 aboutthe moving image obtained by performing the imaging operation once (seeFIG. 1).

Further, as necessary, the CPU 151 draws additional information in thebird's eye-display image 3 based on the speech information 23, thealternative image 27, the link information 28 of the imaginginformation, or the subject information 29 (S22). For example, it may bepermitted to draw an alternative image such as an icon or anillustration in place of the thumbnail image 5, draw a mark in thethumbnail image 5 in which a speech was detected, or draw metadata ofthe thumbnail image 5.

Then, the CPU 151 transmits data of the thus generated bird'seye-display image 3 to the display device 130 so that it may display thebird's eye-display image 3 on it (S24). As a result, the bird'seye-display image 3 such as shown in FIG. 1 appears on the displaydevice 130, thereby permitting the user to appropriately grasp thepicked up contents and imaging situations of the moving image.

[Display Example of Bird's Eye-Display Image]

Next, a description will be given of a display example of the bird'seye-display image 3 according to the present embodiment with referenceto FIGS. 8 to 14. It is to be noted that the bird's eye-display image 3described below will be generated by the display image generation unit60 and displayed on the display device 130 by the display control unit70 in the above imaging device 10.

(1) Combination of Bird's Eye Display and Regular Playback Display (FIG.8)

As shown in FIG. 8, the display control unit 70 provides display of acombination of the bird's eye-display image 3 and the regularplayback-display image 8. If the user selects one thumbnail image 5 cfrom the bird's eye-display image 3, the display control unit 70regularly reproduces the moving image data 21 starting from a time thatcorresponds to the selected thumbnail image 5 c, thus reproducing theregular playback-display image 8 as a moving image. In such a manner, byselecting the thumbnail image 5 c to be confirmed desirably in thebird's eye-display image 3, the user can view the regularplayback-display moving image 8 starting from that position. Thus, theuser can grasp the contents of the picked up moving image moreaccurately.

Further, a video being reproduced and an imaging position in the regularplayback-display image 8 will be displayed in such a manner that theycan be distinguished from the other portions. In the shown example, anarrow-shaped pointer 12 is drawn on the thumbnail image 5 c beingreproduced in regular playback display. This enables the user torecognize that a video of the thumbnail image 5 c and the subsequentimages is being reproduced in regular playback display currently as wellas its reproduction position.

(2) Combination of Bird's Eye Display and Film Roll Display (FIG. 9)

As shown in FIG. 9, the display control unit 70 provides combineddisplay of the bird's eye-display image 3 and the film roll displayimage 9. The thumbnail images 5 a-5 e of the bird's eye-display image 3correspond to thumbnail images 11 a-11 e of the film roll display image9 respectively. If the user selects one thumbnail image 5 c from thebird's eye-display image 3, the display control unit 70 reproduces avideo starting from the position of the thumbnail image 11 ccorresponding to the selected thumbnail image 5 c in the film rolldisplay image 9. In such a manner, by combining the bird's eye-displayimage 3 and the film roll display image 9, it is possible to display thecontents of the picked up moving image, in particular, the informationof the time axis t in a way more easily to understand.

Further, a video being reproduced and an imaging position in the filmroll display image 9 will be displayed in such a manner that they can bedistinguished from the other portions. In the shown example, thethumbnail image 5 a of the bird's eye-display image 3 and the thumbnailimage 11 c in the film roll display image 9 are enclosed by frames 13and 14 respectively. This enables the user to recognize that a video ofthe thumbnail image 5 c and the subsequent images is being reproducedcurrently as well as its imaging position.

(3) Display of Alternative Images and Metadata in Bird's Eye Display(FIG. 10)

As shown in FIG. 10, the display control unit 70 can display analternative image 15 indicative of the contents of imaging situations ofthe thumbnail image 5 such as an icon, an illustration, a still image,or a text in place of the thumbnail image 5 in the bird's eye-displayimage 3. Which one of the thumbnail-size 5 and the alternative image 15should be displayed may be selected by the user arbitrarily orautomatically decided by the display image generation unit 60 inaccordance with the size of the thumbnail image 5.

Now, a description will be given of a specific example of thealternative image 15. For example, an alternative image 15 b may be apartially expanded still image in the thumbnail image 5 b (for example,cutout image of only the face of a person in the thumbnail image 5).Another alternative image 15 c may be a simplified illustration of thecontents of the thumbnail image 5 c (for example, illustration of asubject (stage) as a whole in the thumbnail image 5 c). A furtheralternative image 15 d may be an icon representing the imaging device10. The icon should preferably be drawn in such a manner as to indicatea direction of the imaging device 10 in which it has picked up thethumbnail image 5 b. This enables the user to view the iconizedalternative image 15 c, thereby easily grasping an imaging direction atthat position. Further, as the alternative image 15, a text (not shown)may be displayed which represents the contents of the thumbnail image 5.Such alternative images 15 b, 15 c, and 15 d enable the user to easilygrasp the contents of the thumbnail image 5.

Further, if the thumbnail image 5 is displayed too small in the bird'seye-display image 3, the thumbnail image 5 may be replaced by thealternative image 15. As described above, in the bird's eye-displayimage 3, bird's-eye-view display is given of the virtualthree-dimensional space 6 containing the thumbnail image 5. Accordingly,the display image generation unit 60 changes the sizes of the thumbnailimage 5 and the three-dimensional model 4 to be drawn in accordance withthe drawings positions of the respective thumbnail image 5 andthree-dimensional model 4 in the virtual three-dimensional space 6. Forexample, if the drawing position of the thumbnail image 5 is backward inthe virtual three-dimensional space 6, it is drawn small, and if it isforward, it is drawn large. It is thus possible to provide bird's eyedisplay of the virtual three-dimensional space 6 three-dimensionally.

However, as shown in FIG. 1, the thumbnail images 5 b, 5 c, and 5 d aredisplayed small because they are drawn backward in the virtualthree-dimensional space 6, so that their contents may be difficult tograsp in some cases. Therefore, the display image generation unit 60automatically draws the above alternative images 15 b, 15 c, and 15 d inplace of the thumbnail images 5 b, 5 c, and 5 d to be drawn if thethumbnail images 5 b, 5 c, and 5 d take a predetermined size or less.This gives display of the alternative images 15 b 15 c, and 15 d thatsimply represent the contents of the small thumbnail images 5 b, 5 c,and 5 d. In such a manner, the user can easily grasp the imaged contentsand imaging situations by viewing large the thumbnail images 5 a and 5 edisplayed forward and the easy-to-view alternative images 15 b, 15 c,and 15 d displayed backward in the bird's eye-display image 3.

Next, a description will be given of display of metadata 16 about thethumbnail image 5 with reference to FIG. 10 further. As shown in FIG.10, the display control unit 70 displays various kinds of the metadata16 accompanying the thumbnail image 5 in the bird's eye-display image 3in a condition where the data 16 is related to the thumbnail image 5.The metadata 16 is information that represents the picked up contentsand imaging conditions of the thumbnail image 5. For example, themetadata 16 may come in the above imager information 30 (name, behavior,etc.), the imaging device information 31 (manufacturer's name, type,identification information, camera setting value, operation of theimaging device 10), the subject information 29 (name, size, behavior,environment, etc. of the subject), etc. In the example of FIG. 10, asthe metadata 16 are displayed a manufacturer's “Manufacturer A”, a type“ABC123” of the imaging device 10 and the subject's name “Son andteacher”. By analyzing the moving image data 21 and the thumbnail image5 by using the subject estimation unit 58 and performing facerecognition processing, it is possible to determine a person in thesubject contained in the thumbnail image 5. It is thus possible todisplay the name of a subject in the thumbnail image 5 as the metadata16 of the thumbnail image 5. By displaying such metadata 16 in acondition where it is related to the thumbnail image 5, the user cangrasp the video contents and imaging situations more accurately.

(4) Display of Movement Track Line 7 in Bird's Eye View Mode (FIGS. 11and 12)

As shown in FIG. 11, the display control unit 70 thins the displayinterval of the thumbnail images 5 and, instead, displays movement pathinformation of the imaging device 10. The movement path information ofthe imaging device 10 to be used may be, for example, the movement trackline 7 interconnecting the plurality of thumbnail images 5. The movementtrack line 7 represents a movement track of the imaging device 10 in thevirtual three-dimensional space 6 and is displayed in such a manner asto connect the thumbnail images 5 in order in which they were picked up.The movement track line 7 is one example of information that relates theplurality of thumbnail images 5 to each other in order in which theywere picked up. The display image generation unit 60 draws the movementtrack line 7 based on the movement track information 24 obtained by themovement track sensor 172, for example. In this case, the display imagegeneration unit 60 draws the movement track wine 7 along a movementtrack of the imaging device 10 at the time of actual imaging in such amanner as to connect the thumbnail images 5 in the virtualthree-dimensional space 6 in order in which they were picked up.

For example, a movement track line 7 a connects the thumbnail images 5 aand 5 b and is drawn along the movement track of the imaging device 10at the time when it picked up a moving image of the thumbnail images 5 ato 5 b. Similarly, movement track lines 7 b, 7 c, and 7 d are drawnbetween the thumbnail images 5 b and 5 c, between the thumbnail images 5c and 5 d, and between the thumbnail images 5 d and 5 e respectively.

In such a manner, instead of displaying a lot of the thumbnail images 5in the bird's eye-display image 3, the display interval of the thumbnailimages 5 is thinned, to display the movement track line 7 (pathinformation of the imaging device 10) connecting the thumbnail images 5to each other. It is thus possible to exclude the redundant thumbnailimages 5 and the undesirable ones picked up when the imaging device 10was moving from the bird's eye-display image 3, thereby displaying thethumbnail images 5 in the bird's eye-display image 3 in a way easy tounderstand. Moreover, by connecting the thumbnail images 5 to each otherwith the movement track line 7, it is possible to show order in whichthe thumbnail images 5 were picked up (5 a→5 b→5 c→5 d→5 e) andrepresent their interrelationship. Further, drawing the movement trackline 7 as a straight line, a curved line, or a polygonal line in such amanner as to thread through between the three-dimensional models 4, itis possible to accurately represent a movement path of the imagingdevice 10 at the time when it picked up a moving image. By relating thethumbnail images 5 to each other with such a movement track line 7 inthe bird's eye-display image 3 in order in which they were picked up,the user can grasp the contents and imaging situations of the movingimage more easily and accurately.

Further, how easily the bird's eye-display image 3 can be viewed may beinfluenced by which imaging position in the virtual three-dimensionalspace 6 the thumbnail images 5 are to be displayed at and which imagingposition else in the virtual three-dimensional space 6 the movementtrack line 7 is to be displayed at. For example, if the thumbnail images5 are extracted from the movement image data 21 at a predetermined timeinterval, there is a possibility that some of the thumbnail images 5 mayhave a poor quality. Therefore, in the present embodiment, whenextracting a plurality of the thumbnail images 5 from the movement imagedata 21, such good thumbnail images 5 as to be free of camera shakesbecause they were picked up by the imaging device 10 standing still andsuch thumbnail images 5 as to have a high image quality are extractedselectively. It is thus possible to display the thumbnail images 5(picked up at rest) that provide a basic pint in an imaging scene andthe thumbnail images 5 with a high image quality, thereby improving thedegree of easy-to-view of the bird's eye-display image 3.

Next, a description will be given of how to represent the imagingsituations and video contents by using the movement track line 7 withreference to FIG. 12. In the bird's eye-display image 3, the movementpath information (for example, the movement track line 7) may beappropriately represented using a line type, a text, etc. in accordancewith the operating situations and video contents of the imaging device10 at the time of imaging. For example, as shown in FIG. 12, themovement track line 7 connecting the thumbnail images 5 to each othermay be drawn using different line types with different contents andimaging situations of a moving image between the thumbnail images 5. Thetypes of the movement track line 7 include, for example, a line shape(straight line, dotted line, broken line, knurling line, etc.), a linecolor, a line thickness, blinking/unblinking of the line, etc. Adescription will be given below of examples of changing the line type ofthe movement track line 7 in accordance with the contents and imagingsituations of a moving image, with reference to an example of the bird'seye-display image 3 of FIG. 12.

(a) Example of Changing Line Type in Accordance with Movement Speed ofthe Imaging Device 10 During Photographing

As shown in FIG. 12, the thumbnail images 5 a and 5 b are connected toeach other by displayed solid-line movement track line 7 e anddotted-line movement track line 7 g. The dotted-line movement track line7 g represents the movement speed of the imaging device 10 at the timewhen it has picked up the moving image in the displayed section. Indisplay, for example, the thinner the dotted line is, the slower themovement speed of the imaging device 10 is. Further, the solid-linemovement track line 7 e indicates smooth movement of the imaging device10 in the displayed section. The display image generation unit 60determines the movement speed of the imaging device 10 in each of thesections based on the movement track information 24 (see FIG. 6)representing the movement loci and postures of the imaging device 10,thereby changing the line type of the movement track line 7 inaccordance with the movement speed of the imaging device 10.

(b) Example of Changing Line Type in Accordance with Camera ShakeCondition of Video

As shown in FIG. 12, the thumbnail images 5 d and 5 e are connected toeach other by displayed a knurling-line movement track line 7 f. Theknurling-line movement track line 7 f represents the camera shakecondition of a moving image picked up in the displayed section. Indisplay, for example, the more knurled the line is, the larger thecamera shake is. The display image generation unit 60 determines thecamera shake condition of the moving image in each of the sections basedon the movement track information 24 (see FIG. 6) representing themovement loci and postures of the imaging device 10, thereby changingthe line type of the movement track line 7 in accordance with the camerashake condition.

(c) Example of Changing Line Type (Line Color) in Accordance withSubject in Video

Although not shown, a subject being shown up in a moving image in anygiven section can be represented using the color of the movement trackline 7. For example, the blue movement track line 7 may indicate Mr. A,the red movement track line 7 may indicate Mr. B, and the spottedmovement track line 7 of blue and red may indicate that Mr. A and Mr. Bare both shown up. The display image generation unit 60 can determine asubject contained in a moving image in each of the sections based on thesubject information 29 (see FIG. 6) obtained by analyzing the movingimage (for example, face recognition), thereby changing the color of themovement track line 7 in accordance with the subject.

(d) Example of Changing Line Type (Line Thickness) in Accordance withImportance of Subject in Video

Although not shown, the importance of a subject being shown up in amoving image in any given section can be represented by thickness of themovement track line 7. For example, the thicker the movement track line7 is, the higher the importance is of the subject shown up in the videoin the section. For example, the thick line indicates that Mr. C isshown up. The display image generation unit 60 determines a subjectcontained in a moving image in each of the sections based on the subjectinformation 29 (see FIG. 6) obtained by analyzing the moving image (forexample, face recognition), thereby changing the thickness of themovement track line 7 in accordance with the importance of the subject.The importance of the subject can be preset by the user, for example.

(e) Example of Changing Line Type (Blinking/Unblinking) in Accordancewith Operation Contents of Imaging Device 10

Although not shown, the operations of the imaging device 10 at the timewhen it picked up a moving image in each section can be represented bythe display mode (for example, blinking/unblinking) of the movementtrack line 7. For example, if the movement track line 7 is blinking, itindicates that the lens in the imaging device 10 was zoomed duringimaging of that section. The display image generation unit 60 determinesthe operation contents of the imaging device 10 in each of the sectionsbased on the camera installation values contained in the imaging deviceinformation 31 (see FIG. 6) acquired during imaging, thereby changingthe display mode (blinking/unblinking) of the movement track line 7 inaccordance with the operation contents.

(f) Example of Changing Line Type (Additional Information Appended/notAppended to Line) in Accordance with Sound Contents

Although not shown, it is possible to indicate whether a meaningfulsound is recorded in a moving image in a given section by appending ornot appending the additional information such as, for example, a clef“J” to the movement track line 7. For example, if the clef is added, itindicates that a meaningful sound (for example a man's speech) otherthan noise and environmental sounds is recorded. The display imagegeneration unit 60 determines the sound contents in each of the sectionsbased on the speech information 23 (see FIG. 6) acquired by analyzingsound data, thereby appending the additional information to the movementtrack line 7 in accordance with the sound contents.

As thus described with reference to FIG. 12, in the bird's eye-displayimage 3, the movement track line 7 will be displayed in the various linetypes in accordance with the contents and imaging situations of a movingimage between a plurality of the thumbnail images 5. This enables theuser to easily and appropriately grasp the video contents or imagingsituations in each of the sections in which the moving image was pickedup.

(5) Display of a Plurality of Moving Images in Bird's Eye View Mode(FIGS. 13 and 14)

The above examples have been described with reference to the case ofproviding bird's eye display of a moving image of one file (one movingimage file) imaged and video-recorded by a series of imaging operations(1rec), by using one bird's eye-display image 3. Now, a description willbe given of an example of displaying a plurality of moving images pickedup by the different imaging devices 10 or at different times by usingone bird's eye-display image 17.

As shown in FIG. 13, the display control unit 70 can display thethumbnail images 5 f-5 i of a plurality of moving images picked up bythe different multiple imaging devices 10 (cameras 1-4) at the samelocation. In the shown example, the bird's eye-display image 17 isdisplayed which shows appearances of a field day on the athletic groundof an elementary school. The bird's eye-display image 3 gives bird's eyedisplay of the virtual three-dimensional space 6 containing thethree-dimensional model 4 c of a track on the athletic ground as well asthe thumbnail images 5 f-5 i of four moving images picked up by thedifferent multiple imaging devices 10 (cameras 1-4) on this ground. Ofthese, the three thumbnail images 5 f-5 h were picked up on the fieldday in 2008, while the remaining one the thumbnail image 5 i was pickedup on the field day carried out at the same location in 2007.

It is thus possible to simultaneously display the thumbnail images 5 f-5i of a plurality of moving images picked up at the same location byusing one bird's eye-display image 17, irrespective of differences inthe imaging device 10 or imaging date/time. Thus, even if a plurality ofmoving images are video-recorded by performing a plurality of times ofimaging operations at the same location, the multiple moving images canbe displayed in the one bird's eye-display image 3 in a condition wherethey are related to the imaging location. Therefore, the user canaccurately grasp the contents and imaging situations of thevideo-recorded multiple moving images in the bird' eye-display image 17.Further, they can be compared to moving images video-recorded at thesame location in the past.

Moreover, the display control unit 70 can display the bird' eye-displayimages 17 of the plurality of moving images shown in FIG. 13 and thebird's eye-display image 3 of one such moving image as shown in FIG. 1in a condition where they are switched with each other. For example,suppose that the user has selected one thumbnail image 5 f from among aplurality of the thumbnail images 5 f-Si displayed in the bird'seye-display image 17 of FIG. 13. Then, the display control unit 70displays such a bird's eye-display image 3 as shown in FIG. 1 about themoving image of the selected thumbnail image 5 f in place of the bird'eye-display image 17 of FIG. 13. This enables the user to grasp anoverall image of a plurality of moving image files by viewing the bird'eye-display image 17 of FIG. 13, thereby confirming the details of themoving image files by viewing the bird' eye-display image 3 of FIG. 1.

Further, as shown in FIG. 14, the display control unit 70 can alsocouple a plurality of bird' eye-display images 3 a-3 c about a movingimage picked up at spatially successive locations, thereby displayingone bird' eye-display image 18 that represents the virtualthree-dimensional space 6 having a larger extent. This enables the userto further more easily grasp and manage a plurality of moving imagefiles. By using the composite bird's eye-display images 17 and 18 thatcombine such a plurality of moving image files, a plurality of users cancollectively display the gist of the plurality of moving image filespicked up by the different imaging devices 10 at different times. It isthus possible to easily grasp and manage a lot of moving images.

The above display examples of the bird' eye-display image 3 according tothe present embodiment have been described with reference to FIGS. 8 to14. By using the bird' eye-display image 3 according to the presentembodiment, the user can easily and accurately grasp the picked upcontents and imaging situations of the moving images as a whole pickedup by the imaging device 10.

3. Second Embodiment

Next, a description will be given of the display control apparatus andmethod according to the second embodiment of the present invention.

As shown in FIG. 15, the second embodiment may be an example in whichthe display control apparatus of the present invention has been appliedto an information processing apparatus 200 (for example, a personalcomputer). As a play-back application in the reproduction of a movingimage recorded in a recording medium, the above bird's eye-display image3 is displayed on a display device 209 in the information processingapparatus 200. For example, if the user would like to view or edit thedata of a moving image picked up and video-recorded by the imagingdevice 10, the bird's eye-display image 3 about the data of the movingimage will be displayed on the display device 209. Further, the secondembodiment provides a video reproducing/editing system that adds thefunctions of video reproduction operations, video editing, and videoretrieval to the above video display method that uses the bird'seye-display image 3.

[Hardware Configuration of Information Processing Apparatus]

First, a description will be given of a hardware configuration of theinformation processing apparatus 200 according to the present embodimentwith reference to FIG. 16. FIG. 16 is a block diagram showing thehardware configuration of the information processing apparatus 200according to the present embodiment.

As shown in FIG. 16, the information processing apparatus 200 includes,for example, a CPU 201, an ROM 202, an RAM 203, a host bus 204, a bridge205, a external bus 206, and an interface 207. The informationprocessing apparatus 200 further includes an input device 208, thedisplay device 209, a storage device (HDD) 210, a drive 211, aconnection port 212, and a communication device 213. In such a manner,the information processing apparatus 200 can include a general purposecomputer apparatus (for example, PC).

The CPU 201 functions as an arithmetic processing apparatus and acontrol apparatus, to operate in accordance with various programs, thuscontrolling the units in the information processing apparatus 200. TheCPU 201 performs various kinds of processing in accordance with aprogram stored in the ROM 202 or a program loaded from the storagedevice 210 to the RAM 203. The ROM 202 stores the program and arithmeticparameters used by the CPU 201 and functions as a buffer to reduceaccesses placed to the storage device 210 from the CPU 201. The RAM 203temporarily stores the program used by the CPU 201 when it performs theprocessing and the parameters etc. which changes appropriately duringthe performance. Those are connected to each other by the host bus 204constituted of a CPU bus etc. The host bus 204 is connected to anexternal bus 206 such as a peripheral component interconnect/interface(PCI) bus via the bridge 205.

The input device 208 includes, for example, operation means such as amouse, a keyboard, a touch panel, a button, a switch, or a lever and aninput control circuit to generate an input signal and send it to the CPU201. The display device 209 includes, for example, a liquid crystaldisplay (LCD), an organic EL display, a plasma display, a cathode raytube (CRT) display etc. Further, it is equipped also with a sound outputdevice (not shown) such as a speaker.

The storage device 210 is used to store various kinds of data andconstituted of, for example, an external device such as an HDD or abuilt-in disk drive. The storage device 210 drives a hard disk, which isa recording medium, to store in it the program executed by the CPU 201and various kinds of data. The drive 211 is used to carry out read andwrite operations to the recording medium and built in or externallymounted to the information processing apparatus 200. The drive 211performs write/read operations of various kinds of data to a removablerecording medium such as, for example, a magnetic disc, an optical disc,a magneto-optical disc, or a semiconductor memory loaded into theinformation processing apparatus 200.

The connection port 212 is used to connect an external peripheral deviceand has a connection terminal such as, for example, a USB or anIEEE1394. The connection port 212 is connected to the CPU 201 etc. viathe interface 207, the external bus 206, the bridge 205, or the host bus204. The communication device 213 is a communication interfaceconstituted of, for example, a communication device to connect to anetwork. The communication device 213 transmits various kinds of data toand receives them from an external device having network communicationfunctions, via the network.

For example, the information processing apparatus 200 can acquire movingimage data 21 and its additional information from the imaging device 10having the network communication functions or any other device, via thenetwork. Further, the information processing apparatus 200 can acquirethe moving image data 21 obtained by the imaging device 10 and itsadditional data etc. from a recording medium recording the data.

[Functional Configuration of Information Processing Apparatus]

Next, a description will be given in detail of the functionalconfiguration of the information processing apparatus 200 according tothe present embodiment with reference to FIG. 17. FIG. 17 is a blockdiagram showing the functional configuration of the informationprocessing apparatus 200 according to the present embodiment. It is tobe noted that FIG. 17 schematically shows the functional units that areused to display the bird's eye-display image 3, which features thepresent embodiment.

As shown in FIG. 17, the information processing apparatus 200 includes astorage unit 20, a preprocessing unit 50, a display image generationunit 60, the display control unit 70, an image processing unit 80, thedisplay device 209, and the input device 208. Of these, thepreprocessing unit 50, the display image generation unit 60, the displaycontrol unit 70, and the image processing unit 80 are realized by, forexample, software installed in the information processing apparatus 200.That is, for example, the CPU 201 shown in FIG. 16 operates inaccordance with the program stored in the ROM 202 etc., therebyfunctioning as the preprocessing unit 50, the display image generationunit 60, the display control unit 70, and the image processing unit 80.It is to be noted that the program can be provided to the informationprocessing apparatus 200 via a recording medium or a network.

In contrast to the above imaging device 10 (FIG. 5) according to thefirst embodiment, the information processing apparatus 200 (FIG. 17)according to the second embodiment is not equipped with the imagingprocessing unit 40, the distance sensor 170 nor the movement tracksensor 172 but includes the image processing unit 80. Except for thedifference, the second embodiment is the same as the first embodiment,and repetitive description will be omitted on the identical componentsof the storage unit 20, the preprocessing unit 50, the display imagegeneration unit 60, and the display control unit 70.

As shown in FIG. 17, the information processing apparatus 200 acquiresmoving image data 21, imager information 30, imaging device information31, and sensor-measured data 32 via a recording medium or a network fromany other device such as the imaging device 10 and stores them in thestorage unit 20. The sensor-measured data 32 corresponds to datameasured by the distance sensor 170 or the moving track sensor 172described in FIG. 6.

Similar to the first embodiment, the preprocessing unit 50 generatesvarious kinds of information and data pieces 22-29 based on input datasuch as the moving image data 21 and the sensor-measured data 32. It isto be noted that if the information processing apparatus 200 can acquirethose information and data pieces 22-29 from external device such as theimaging device 10, the preprocessing unit 50 is unnecessary.

Further, if the sensor-measured data 32 may not be obtained from theimaging device 10, a movement track estimation unit 53 in thepreprocessing unit 50 may analyze the moving image data 21, to estimatea movement track, a posture, etc. of the imaging device 10 at the timeof imaging. Further, a three-dimensional shape estimation unit 54 mayanalyze the moving image data 21, to identify and extract an objectcontained in a video represented by the moving image data 21, therebyestimating its three-dimensional shape, position, etc.

Similar to the first embodiment, the display image generation unit 60generates a bird's eye-display image 3, which is then displayed by thedisplay control unit 70 on the display device 209.

The image processing unit 80 processes (edits, generates, retrieves,etc.) an image such as the moving image data 21 in response to an inputoperations by the user on the input device 208. The image processingunit 80 includes an image editing unit 82 and an image retrieval unit84.

The image editing unit 82 edits the moving image data 21 in response touser operations on a three-dimensional model 4 or a thumbnail image 5 inthe bird's eye-display image 3 shown on the display device 209. Thisenables the user to edit a video only by dragging and dropping theedit-target three-dimensional model 4 or thumbnail image 5 on the bird'seye-display image 3, thus facilitating the editing job. The details ofsuch editing processing will be described later (see FIGS. 18 and 19).

Further, the image editing unit 82 generates an image based on themoving image data 21 in response to user operations on the bird'seye-display image 3 shown on the display device 209. For example, if anangle of the moving image is specified by the user in the bird'seye-display image 3, the image editing unit 82 processes the movingimage data 21 in accordance with the specified angle, thus generatingthe data of the moving image as viewed at the angle. The details of suchangle changing processing will be described later (see FIG. 20).

The image retrieval unit 84 retrieves any other three-dimensional model4 which is the same as or similar to that selected by the user from thebird's eye-display image 3 shown on the display device 209 or an imagecontaining the three-dimensional model 4. The image retrieval unit 84can retrieve the three-dimensional model 4 by specifying not only itsshape but also its color and size. Further, the image retrieval unit 84can perform retrieval by use of the three-dimensional model 4 among aplurality of pieces of the moving image data 21. The details of suchretrieval processing will be described later (see FIG. 21).

[Display Example of Bird's Eye-Display Image]

Next, a description will be given of a display example of the bird'seye-display image 3 according to the present embodiment with referenceto FIGS. 18 to 23. Moving images video-recorded by a recorder such asthe imaging device 10 have commonly been viewed, edited, and retrievedusing the information processing apparatus 200. In the following, adescription will be given of the display examples of applying the bird'seye-display image 3 to the viewing, editing, and retrieval of suchmoving images.

(1) Video Editing to Add/Delete Image (FIG. 18)

As shown in FIG. 18, the display control unit 70 displays the bird'seye-display image 3 about a given moving image as well as icons 220 and221 indicative of additional images and icons 230 and 231 indicative ofselection or deletion of a scene on the display device 209. Theadditional images may be any one of a moving image and a still image.

First, a description will be given of additional editing of an image tothe existing moving image. The user can edit the moving image bydragging and dropping the icons 220 and 221 indicative of the images ofa scene to be added desirably, onto the thumbnail image 5 in the bird'seye-display image 3. When the post-editing moving image ids reproduced,the images of the added scene will be reproduced in order in which theywere added.

Describing in more detail, the example shown in FIG. 18 displays theicon 220 indicative of the image of an opening scene and the icon 221indicative of the image of an ending scene on the left side of thebird's eye-display image 3. The user drags and drops the icon 220 of theopening scene onto the top thumbnail image 5 a and the icon 221 of theending scene onto the last thumbnail image 5 a. In response to such useroperations, the image editing unit 82 adds the image of the openingscene to the top of the moving image given by the thumbnail images 5 a-5e and the image of the ending scene to the end of the moving image. Insuch a manner, the user can edit a video by performing simple operationsin the bird's eye-display image 3.

Next, a description will be given of selecting and deleting of a scenein a moving image. The user can edit a moving image by dragging anddropping the thumbnail image 5 in the bird's eye-display image 3 ontothe icons 230 and 231 indicative of scene selection and deletionrespectively. When the post-editing moving image is reproduced, theselected scenes will be reproduced, but not the deleted scenes.

Describing in more detail, the example shown in FIG. 18 displays theicon 230 indicative of scene selection and the icon 231 indicative ofscene deletion on the right side of the bird's eye-display image 3. Theuser selects a scene containing the thumbnail image 5 c by dragging anddropping the thumbnail image 5 c onto the selection icon 230. Further,the user deletes a scene containing the thumbnail image 5 c by draggingand dropping the thumbnail image 5 c onto the deletion icon 231. Inresponse to such user operations, the image editing unit 82 edits theoriginal moving image in such a manner that the scene containing thethumbnail image 5 c may be left or deleted. By such scene selection anddeletion by use of the thumbnail image 5, the moving image in apredetermined section containing the thumbnail image 5 is selected ordeleted.

In such a manner, the user can edit a video by selecting and deletingscenes of a moving image through simple operations on the bird'seye-display image 3. Further, after editing, in the bird's eye-displayimage 3, a movement track line 7 corresponding to the deleted scene maybe expressed as a dotted line or the thumbnail image 5 of the scene maybe made semitransparent. This enables the user to recognize that somescenes in the moving image have been deleted.

(2) Video Editing by Use of Three-Dimensional Model (FIG. 19)

As shown in FIG. 19, the display control unit 70 causes the displaydevice 209 to display the bird's eye-display image 3 about a givenmoving image as well as an icon 240 indicative of selection or deletionof the three-dimensional model 4 and an icon 241 indicative of an objectof a three-dimensional model to be added desirably.

First, a description will be given of video editing by use of deletionof the three-dimensional model 4 from the bird's eye-display image 3.The user can edit a moving image by dragging and dropping athree-dimensional model 4 d in the bird's eye-display image 3 onto theicon 240 indicative of three-dimensional model deletion. In response tosuch user operations, the image editing unit 82 deletes the imageportion of an object corresponding to the deleted three-dimensionalmodel 4, to supplement an appropriate image by utilizing images aroundthe deleted portion and those before and after the deletion. As aresult, when the post-editing moving image is reproduced, the image ofthe object (right backward table in the example of the figure)corresponding to the deleted three-dimensional model 4 d is not shownand the resultant empty portion is filled with a drawing of theappropriate image by use of the surrounding time-wise and space-wiseinformation. By thus editing a video utilizing the three-dimensionalmodel 4, persons and objects undesirably imaged during photographing canbe easily selected and deleted.

Next, a description will be given of video editing by use of addition ofthe three-dimensional model 4 to the bird's eye-display image 3. Theuser can edit a moving image by dragging and dropping the icon 241indicative of the object of a three-dimensional model to be addeddesirably onto any given portion in the bird's eye-display image 3. Inresponse to such user operations, the image editing unit 82 draws animage (Christmas tree in the shown example) of the object correspondingto the added three-dimensional model 4 to a specified position. As aresult, when the post-editing moving image is reproduced, the imagecorresponding to the added three-dimensional model 4 is drawn to theposition where the icon was dropped so that it may be added, therebyavoiding drawing of a background image of a portion hidden by the addedimage.

Next, a description will be given of video editing by use of movement ofthe three-dimensional model 4. The user can edit a moving image bydragging and dropping an arbitrary three-dimensional model 4 a in thebird's eye-display image 3 onto any given portion of the bird'seye-display image 3. In response to such user operations, the imageediting unit 82 deletes an image (upper left table in the shown example)of the object corresponding to the moved three-dimensional model 4 a atthe original position and draws it to a movement-destination position.At the original position, the image editing unit 82 supplements anappropriate image by utilizing images around the deleted portion andthose before and after the deletion. As a result, when the post-editingmoving image is reproduced, the object of the three-dimensional model 4a is drawn to the movement-destination position, but not to the originalposition.

In such a manner, the user can edit a video by adding, deleting, andmoving the three-dimensional model 4 through simple operations on thebird's eye-display image 3.

(3) Video Editing by Means of Angle Specification (FIG. 20)

As shown in FIG. 20, the user can adjust an angle-specification arrow250 in the bird's eye-display image 3, thereby specifying an angle(arrow 251) different from an angle (arrow 250) at the time when amoving image was picked up. The angle means a camera angle (viewpoint)at which a subject was imaged. In accordance with the specified angle,the image editing unit 82 performs image processing on the moving imagedata 21, thereby generating data of the moving image as viewed at theangle. As a result, when the video is reproduced, the moving image (theregular playback-display image 8) viewed at the specified angle isgenerated and displayed. In such a manner, the user can easily specify adesired angle so that the information processing apparatus 200 cangenerate and display a moving image having the angle as its viewpoint.By thus viewing a moving image at a different angle, the user can moreaccurately grasp the contents and imaging situations of the movingimage.

(4) Image Retrieval by Use of Three-Dimensional Model (FIG. 21)

AS shown in FIG. 21, the user can select any given three-dimensionalmodel 4 a in the bird's eye-display image 3 and retrievethree-dimensional models 4 e-4 i that are the same as or similar to thethree-dimensional model 4 a. In more detail, first the user selects onethree-dimensional model 4 a in the bird's eye-display image 3. Then, theimage retrieval unit 84 displays the selected three-dimensional model 4a in a retrieval window 261. If the user presses a retrieval button 262,the image retrieval unit 84 retrieves the three-dimensional models 4 e-4i similar to the selected three-dimensional model 4 a and furthersearches an entirety of the moving image for an image portion containingobjects represented by the retrieved three-dimensional models 4 e-4 i.Thus, the user can easily retrieve a video in which a desired object(subject) is shown up.

Further, by specifying not only the shape of the three-dimensional model4 but also its color and size, it is possible to retrieve an imagecontaining similar three-dimensional models 4. It is also possible toretrieve similar images by using the three-dimensional model 4 across aplurality of moving images.

Further, the user can select any given three-dimensional model 4 in thebird's eye-display image 3 and add meta-information to thethree-dimensional model 4. The image retrieval unit 84 can retrieve thethree-dimensional model 4 based on that meta-information, therebyretrieving images that contain the three-dimensional model 4. It is alsopossible to retrieve similar images by using the meta-information of thethree-dimensional model 4 across a plurality of moving images.

As described above, the bird's eye-display image 3 shows thethree-dimensional model 4 of an object in a moving image so that thethree-dimensional model 4 can be suitable utilized as a retrievalcondition. By retrieving a similar image or a related image by using thethree-dimensional model 4 as the retrieval condition, the user caneasily and quickly find out a desired image.

(5) Bird's Eye Display of Subject Containing Movable Body (FIGS. 22 and23)

FIG. 22 shows the bird's eye-display image 3 in a case where a subjectcontaining a movable body (for example, train) is imaged from a fixedpoint. In this case, the movable body may be, for example, the shownvehicle (for example, train), an animal, a person, etc. The displaycontrol unit 70 displays a three-dimensional model 4 j of the movablebody and a three-dimensional model 4 k of any other object (for example,elevated bridge) as well as the bird's eye-display image 3 containingone thumbnail image 5 j photographed at an arbitrary time. In this case,since the movable body moves in the moving image as time elapses, anissue occurs on which position in the virtual three-dimensional space 6the three-dimensional model 4 j of the movable body should be drawn at.

In the shown example, the display image generation unit 60 draws thethree-dimensional model 4 j of the movable body to the position at agiven imaging time, to draw the thumbnail image 5 j at the imaging time.In this case based on the moving image as a whole, the display imagegeneration unit 60 fills the movable body's portion 271 not shown up inthe thumbnail image 5 j at the imaging time, thus drawing thethree-dimensional model 4 j of the movable body. A portion 272 enclosedby a slid-line ellipsoid in FIG. 22 is the filled portion in thethree-dimensional model 4 j of the movable body. In such a manner, theuser can appropriately grasp the contents of the moving image picked upfrom the fixed point, especially, the position and the size of themovable body.

Further, as shown in FIG. 23, the user can control the reproduction timein the regular playback-display image 8 by selecting three-dimensionalmodel 4 j of the movable body in the bird's eye-display image 3 using apointer 273 and dragging (that is, moving) the three-dimensional model 4j.

That is, if a movable body is imaged in such a manner, there is anassociation between the position of the movable body and thereproduction time in the moving image. Accordingly, if thethree-dimensional model 4 j of the movable body is moved in the bird'seye-display image 3 by the user, the display control unit 70 controlsthe reproduction time, reproduction direction, and reproduction speed inthe regular playback-display image 8 in accordance with the position,movement direction, and movement speed of the three-dimensional model 4j. In the example of FIG. 23, if the three-dimensional model 4 j ismoved from the right to the left, the regular playback-display image 8is reproduced in as positive reproduction direction. By thusinterlocking movement of the three-dimensional model 4 of the movablebody and regular playback display with each other, the user can provideregular playback display of the moving image at a desired reproductiontime, in a desired direction, and at a desired reproduction speedthrough simple operations, thereby appropriately grasping the contentsof that moving image.

EFFECTS

Thus, the display control apparatus and display method for displaying abird's eye-display image according to the first and second embodimentsof the invention has been described. The regular playback display inrelated art (FIG. 2) and film roll display (FIG. 3) have had an issuethat they would require a lot of time and labor to grasp the picked upcontents of a moving image and, further, could not grasp the imagingsituations.

In contrast, according to the bird's eye-display image 3 according tothe present embodiments, the three-dimensional model 4 of an object isdrawn to its actual position in the virtual three-dimensional space 6and to a position and in a direction at the time when a plurality of thethumbnail images 5 were picked up. Moreover, as information to relatethe thumbnail images 5 to each other, the movement track line 7interconnecting the plurality of thumbnail images 5 in order in whichthey were picked up is drawn along a movement track of the imagingdevice 10. Then, the bird's eye-display image 3 provides bird's eyedisplay of the virtual three-dimensional space 6 containing thethumbnail images 5 and three-dimensional model 4, thus making it easy tograsp the picked up contents and imaging situations by utilizing themovement path of the imaging device 10 and the association between thethumbnail images 5.

Accordingly, the user can easily and instantaneously grasp not only thecontents of an imaged video (moving image) but also the imagingsituations. Thus, the user can efficiently classify, arrange, and edit aplurality of videos recorded by the imaging device 10. As a result, itis possible to overturn the status quo in which a lot of videos arestocked at the user so that the user may not know what to do with them.

Further, besides such easy grasping of video contents, in video editingalso, the user can very easily and quickly reproduce, retrieve, and editthe videos through simple operations such as drag-and-drop operations.Therefore, the even users who have hesitated editing because it istroublesome can utilize the present invention readily.

The future progress of the imaging device can surely be expected to aimat the addition of a function to acquire three-dimensional information,while the video display method according to the present embodiments hasa very high affinity with the function. Further, a recent-year progressin information processing apparatuses and image analyzing technologieshas improved a capability of automatically extracting thethree-dimensional information from a video. Therefore, the video displaymethod according to the present embodiments can accommodate also theassets of videos in related art whose three-dimensional information isyet to be acquired at the time of photographing.

It should be understood by those skilled in the art that variousmodifications, combinations, sub-combinations and alterations may occurdepending on design requirements and other factors insofar as they arewithin the scope of the appended claims or the equivalents thereof.

For example, the above embodiments have applied the display controlapparatus to the imaging device 10 (digital video camera) or theinformation processing apparatus 200 (PC), the present invention is notlimited to them. For example, of course, the display control apparatusof the present invention can be applied to a variety of imaging devicesincluding a digital still camera and a monitor camera and any givendevices having an imaging function such as a cell phone with a cameraand the like. Further, it can be applied to any given electronic devicesas far as they have a video reproduction function although it has noimaging functions. They may include a video recording/reproductionapparatus such as a blu-ray disc/DVD/HDD player, a personal digitalassistant (PDA), a portable video player, a game machine, and a TVreceiver.

Further, the above embodiments have been described with reference to thecase where the individual user would photograph personal contents(self-recording video), the bird's eye display method of the presentinvention can be applied to a wide range field including the productionof a broadcast programs and video media.

Further, the device for displaying the bird's eye-display image 3 of thepresent invention is not limited to two-dimensional monitors. Forexample, by using a three-dimensional monitor, the bird's eye-displayimage can be reproduced in a more steric manner.

Further, although the above embodiments have displayed the movementtrack line 7 as the information to relate the thumbnail images 5 to eachother in order in which they were picked up, the present invention isnot limited to it. For example, as this information, text informationindicative of the order in which the thumbnail images 5 have bee pickedup may be displayed or the marks and icons may be displayed in such amanner as to move along a movement pass of the imaging device 10 in thebird's eye-display image 3.

The present application contains subject matter related to thatdisclosed in Japanese Priority Patent Application JP 2008-264112 filedin the Japan Patent Office on 10 Oct. 2008, the entire content of whichis hereby incorporated by reference.

What is claimed is:
 1. A display control apparatus comprising: a storageunit that stores a moving image picked up by performing successiveimaging operations once by using an imaging device; a display imagegeneration unit that draws a plurality of thumbnail images extractedfrom the moving image to the respective positions of the imaging deviceat the time when the thumbnail images were picked up in a virtualthree-dimensional space in which a three-dimensional model of an objectcontained in the moving image is disposed and draws information thatrelates the thumbnail images in order in which they were picked up inthe virtual three-dimensional space, thereby generating a bird'seye-display image which provides a bird's-eye-view drawing of thevirtual three-dimensional space containing the three-dimensional modeland the thumbnail images; and a display control unit that causes adisplay unit to display the bird's eye-display image.
 2. The displaycontrol apparatus according to claim 1, wherein the storage unit storesmovement track information which represents a movement track of theimaging device at the time when the imaging device picked up the movingimage, as additional information of the moving image and the displayimage generation unit draws the thumbnail images in the virtualthree-dimensional space at the respective positions of the imagingdevice at the time when the thumbnail images were picked up based on themovement track information.
 3. The display control apparatus accordingto claim 2, wherein the information which relates the thumbnail imagesin order in which they were picked up is a movement track linerepresenting the movement track of the imaging device at the time whenthe imaging device picked up the moving image, and the display imagegeneration unit draws the movement track line in the virtualthree-dimensional space in such a manner as to connect the thumbnailimages in the order in which the thumbnail images were picked up basedon the movement track information.
 4. The display control apparatusaccording to claim 3, wherein the display image generation unit drawsthe movement track line connecting the thumbnail images in various linetypes in accordance with contents or imaging situations of the movingimage between the thumbnail images.
 5. The display control apparatusaccording to claim 1, wherein the display image generation unit draws analternative image which represents the contents or imaging situations ofat least some of the thumbnail images to the drawing positions of thethumbnail images in place of the thumbnail images.
 6. The displaycontrol apparatus according to claim 5, wherein the display imagegeneration unit: changes a size of the thumbnail images to be drawn inaccordance with the drawing positions of the thumbnail images in thevirtual three-dimensional space; and if the size of the thumbnail imagesto be drawn is a predetermined size or less, draws the alternative imagein place of the thumbnail images.
 7. The display control apparatusaccording to claim 1, wherein the display image generation unit drawsmetadata about the thumbnail images in the virtual three-dimensionalspace in a condition where the metadata is related to the thumbnailimages.
 8. The display control apparatus according to claim 1, furthercomprising an image editing unit that edits the moving image in responseto user operations on the thumbnail images or the three-dimensionalmodel in the bird's eye-display image shown on the display unit.
 9. Thedisplay control apparatus according to claim 1, further comprising animage editing unit that generates, in response to specification of anangle in the bird's eye-display image shown on the display unit, themoving image as viewed at the specified angle.
 10. The display controlapparatus according to claim 1, further comprising an image retrievalunit that retrieves the image containing the three-dimensional modelwhich is the same as or similar to the three-dimensional model selectedfrom the bird's eye-display image shown on the display unit or an objectcorresponding to the three-dimensional model.
 11. A display controlmethod comprising the steps of: drawing a plurality of thumbnail imagesextracted from a moving image picked up by performing successive imagingoperations once by using an imaging device to the respective positionsof the imaging device at the time when the thumbnail images were pickedup in a virtual three-dimensional space in which a three-dimensionalmodel of an object contained in the moving image is disposed and drawsinformation that relates the thumbnail images in order in which theywere picked up in the virtual three-dimensional space, therebygenerating a bird's eye-display image which provides a bird's-eye-viewdrawing of the virtual three-dimensional space containing thethree-dimensional model and the thumbnail images; and causing a displayunit to display the bird's eye-display image.
 12. A program for causinga computer to perform the steps of: drawing a plurality of thumbnailimages extracted from a moving image picked up by performing successiveimaging operations once by using an imaging device to the respectivepositions of the imaging device at the time when the thumbnail imageswere picked up in a virtual three-dimensional space in which athree-dimensional model of an object contained in the moving image isdisposed and draws information that relates the thumbnail images inorder in which they were picked up in the virtual three-dimensionalspace, thereby generating a bird's eye-display image which provides abird's-eye-view drawing of the virtual three-dimensional spacecontaining the three-dimensional model and the thumbnail images; andcausing a display unit to display the bird's eye-display image.